Tokenization — Field Foundations

The one-sentence version: A language model has never seen a word, a letter, or a sentence. It only ever sees numbers that stand for chunks of text — and tokenization is the step that chops your text into those chunks. Get this, and a dozen weird model behaviors suddenly make sense.

Section 0

Why we start here

Everything else in this part — embeddings, attention, the whole transformer — operates on tokens. If you don't know what a token is, every later diagram is built on a word you're guessing at. So this is the floor. It's also genuinely easy, and it explains some famous model failures, so it's a fun place to start.

By the end you'll be able to explain

what a token is · why models don't use whole words or single letters · how the model decides its chunks (BPE) · what a "token ID" is · and why "strawberry has how many R's?" goes wrong.

Section 1

The model reads in pieces called tokens

Picture a box of LEGO bricks. Not infinite bricks — a fixed box, maybe 50,000 different brick shapes. A language model builds every sentence it ever reads or writes out of that one fixed box of bricks. Those bricks are called tokens.

A token is usually a piece of a word — a "subword." Here's how everyday text gets bricked up:

Text	Tokens (the bricks)
cat	cat — one common word = one brick
cats	cats
tokenization	tokenization — rarer = several bricks
strawberry	strawberry
hello world	hello␣world — ␣ marks the space, and it rides on the FRONT of the next brick

Two things to notice right away:

A common word is often a single brick. A rare word gets split into several.
The space before a word usually sticks to the front of the next token (␣world, not world — ␣ is how a space is drawn here, because an invisible character is impossible to point at). "Start of a new word" is part of the brick.

The wrong turn The intuition almost everyone arrives with is that the model reads letters, or reads words — that somewhere inside there is a version of your sentence as text. There isn't. By the time anything happens, your sentence has already become a list of numbers standing for chunks, and the letters are gone. Almost every confusing thing a model does with spelling, counting or arithmetic starts here.

Analogy — the LEGO box

The model can build anything, but only out of the bricks in its box. If a shape isn't in the box, it builds that shape from smaller bricks it does have. There is no "make a brand-new brick mid-sentence" option. The box is fixed before training even starts.

Section 2

Why not whole words? Why not single letters?

This is the heart of the module. Tokenization is a compromise between two bad extremes.

Extreme A — one word = one brick

The box explodes. Millions of words once you count names, slang, typos, other languages.
New words break it. "Rizz," a brand launched tomorrow, a username — no brick exists, it chokes.
Typos are strangers. "teh" and "the" become totally unrelated bricks. Wasteful.

Extreme B — one letter = one brick

Sequences get enormous. "tokenization" = 1 idea but 12 bricks to haul around. Slow.
Meaning gets shredded. The letter t means almost nothing alone; the model rebuilds every concept from alphabet soup.

The Goldilocks answer — subword tokens

Keep common chunks whole (theingtionstraw) and only split when you hit something rare. Box stays a sane size (~50k–100k bricks), every possible string can still be built (worst case, fall back to letters), most common ideas are 1–2 bricks. That's tokenization.

The trade-off in one line

words → box too big + can't handle new text · letters → sequences too long + meaning shredded · subwords → just right.

Subword units were introduced for neural translation by Sennrich, Haddow & Birch [1]:

+1.1BLEU · EN→DE

+1.3BLEU · EN→RU

Section 3

How the model picks its chunks — Byte-Pair Encoding (BPE)

Who decides which chunks become bricks? An algorithm, run once, before training, over a giant pile of text. The most common one is Byte-Pair Encoding (BPE). The whole idea is one rule, repeated:

BPE's only move

Find the pair of bricks that sits next to each other most often, and glue them into one new brick. Repeat thousands of times.

Let's watch it on a tiny example. Say our entire training text is these four words, and we start with every word split into letters:

Start — everything is letters

playingplaying

runningrunning

jumpingjumping

singingsinging

Round 1 — the pair in is most common → glue → in

playingplaying

runningrunning

jumpingjumping

singingsinging

Round 2 — now ing is everywhere → glue → ing

playingplaying

runningrunning

jumpingjumping

singingsinging

Look what just happened: ing is now a single brick. The algorithm discovered, with zero grammar knowledge, that "-ing" is a useful chunk of English — purely because it's frequent. Run thousands more rounds over the whole internet and you naturally get bricks for thetionstrawberry, common names, and so on. Frequent stuff becomes whole bricks; rare stuff stays in small pieces.

Analogy — texting shorthand

Nobody designed "lol," "brb," "idk." They got typed so often the chunk fused into one unit everybody recognizes. BPE is that, automated: the chunks you use most fuse into single bricks.

The wrong turn The intuition almost everyone arrives with is that the chunks correspond to something linguistic — syllables, or prefixes and suffixes, or units of meaning. They often look like that, which is exactly what makes it convincing. They aren't. They are whatever turned out to be frequent, and frequency and grammar only agree by coincidence.

Key takeaway

The vocabulary (the box of bricks) is learned from frequency, not from a dictionary. That's why token boundaries sometimes look weird to a human — they follow statistics, not grammar.

The algorithm itself is older than the language models that use it: Philip Gage published byte-pair encoding as a compression method in 1994, and Sennrich et al. re-purposed it for tokenization [1]. SentencePiece [2] later added the ability to train the vocabulary from raw sentences rather than from text someone had already split into words — which especially helps languages that don't put spaces between words at all, like Japanese, Chinese and Thai, where "split on spaces first" isn't a step you can even perform.

1994Gage's algorithm, reused

Section 4

From brick to number — the token ID

The model can't do math on the letters of a brick. So once the box is fixed, every brick gets a number — its token ID. The box is just a big numbered lookup table:

"straw"→19587

" world"→1917

"ing"→278

"cat"→9246

So the real journey of your text is:

"playing"→ playing→ 1991278

your text → tokens (bricks) → token IDs (numbers)

That list of numbers — 1991278 — is the only thing the model actually receives. Not the word. Not the letters. Just the ID numbers standing for the chunks. Everything from Module 2 onward starts from these numbers.

“The model doesn't read your sentence. It reads a list of part-numbers for the bricks your sentence was built from.”

Section 5

What real tokenization looks like

A rough feel for it (exact splits vary by model):

The cat sat on the mat. → about 7 tokens (each common word ≈ 1 brick, plus the period).
antidisestablishmentarianism → many bricks (rare → shattered).
Rule of thumb: ~1 token ≈ ¾ of an English word, so ~100 tokens ≈ 75 words.

Common = cheap (few bricks). Rare = expensive (many bricks). Hold onto that — it's the key to the next section.

Section 6

Why you should care — the behaviors this explains

This is where the module pays off. All of these show up in real papers and real model quirks, and they all trace back to "the model only sees tokens."

🍓

"How many R's in strawberry?"

The model never saw s-t-r-a-w-b-e-r-r-y. It saw two bricks, strawberry, as two ID numbers. Counting letters is like being handed two face-down puzzle pieces and asked how often a color appears in the full picture. The info isn't in front of it.

💸

Rare words + typos cost more

"The" is one token; a weird username or long technical term is many. Models are billed and limited per token, so rare text literally costs more and eats more capacity.

📏

"Context window" is in tokens

"128k context" = 128,000 tokens ≈ 96,000 words, not 128,000 words. Tokens are the unit of pricing, limits, and speed.

🌍

Some languages cost more

The box is built from English-heavy text, so English gets efficient whole-word bricks. Many other languages (and emoji, and code) shatter into more pieces — same sentence, more tokens, more money. Petrov et al. [3] measured this across 17 languages and found length disparities of up to 15×, still over 4× at byte level. A real fairness issue researchers write about.

15×worst disparity

4×+still, at byte level

17languages measured

🔢

Arithmetic gets weird

1234 might be one brick, 1235 two. The model isn't seeing tidy digits — it's seeing oddly-chopped number-bricks, part of why raw LLMs fumble math.

🧵

The through-line

Every one of these is the same fact in a different hat: the model's whole reality is the token stream, and the token stream is lossy. It threw away the letters to gain efficiency.

Section 7

The handoff to Module 2

You now have your text turned into a list of ID numbers, one per brick. But a bare ID like 278 carries no meaning yet — it's just a name tag, as arbitrary as a coat-check number. Brick #278 being "near" #279 in the table means nothing.

Module 2 (Embeddings) fixes that: it turns each ID number into a list of values — a vector — that actually places the brick's meaning in space, so king and queen end up near each other and king and banana don't. That's where "meaning" first enters the machine.

🧠 Recap

The whole module in six lines

The model only ever sees tokens (subword chunks), turned into ID numbers. Never words, never letters.
Words as bricks → box too huge + can't handle new text. Letters → sequences too long + meaning shredded. Subwords → the Goldilocks middle.
BPE builds the box with one repeated move: glue the most frequent adjacent pair into a new brick. (We watched i+n → in → ing.)
The box is learned from frequency, not a dictionary — boundaries follow statistics, not grammar.
Each brick → a token ID; the list of IDs is the only input the model gets.
This single fact explains strawberry's R's, token-cost, the context-window unit, cross-language cost, and wonky arithmetic.

✅ Now go and use it

Read it, write your own notes, then talk it through

Step 1 — talk it through. This copies a prompt with the whole module already inside it. Paste it into any AI assistant: it asks you questions one at a time, answers the ones you wrote down, and explains why a wrong answer was tempting. It will not test you.

Step 2 — only when you feel ready. The closed-book gate. It carries this module's own questions and tells the assistant to score you strictly and explain what you misunderstood. Write your own notes first — the notes are for encoding, not for shrinking the test, and the questions cover the whole module, not just what you wrote down.

Download the .md

a read b your own notes c discuss d scored check e on to Module 2

Any assistant works — Claude, ChatGPT, Gemini, an open-source model you run yourself. Free tiers are fine. You can dictate your answers instead of typing.

Sources

Rico Sennrich, Barry Haddow, Alexandra Birch, Neural Machine Translation of Rare Words with Subword Units, 2016 — arXiv:1508.07909. Introduced subword units for neural translation, adapting Philip Gage's 1994 byte-pair-encoding compression algorithm to the job.
Taku Kudo, John Richardson, SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, 2018 — arXiv:1808.06226.
Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, Adel Bibi, Language Model Tokenizers Introduce Unfairness Between Languages, 2023 — arXiv:2305.15425.

← back to the index Free · no account · no tracking